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Abstract: 



This paper focusses on the mechanics of assessing the impact of educational 
technology. The basics of program evaluation, research design, data coilection and 
analysis, and report generation are covered in this paper. This information is usefui to 
personnel responsible for conducting program evaluation research as well as to 
decision makers who are the usual recipients of evaluation reports. 

Terms : 

There are two widely used terms in evaluation research which are rarely defined 
and therefore often misunderstood: formative evaluation and summative evaluation. In 
this paper, formative evaluation is best thought of as an evaluation of "processes" (i.e., 
assessing the status of a district’s technology plan), and summative evaluation is 
essentially an evaluation of "products" (i.e., assessing the impact of technology on 
student learning, performance, and productivity). Each of these models has its own 
utility and purpose, as theoretically, each measures a discrete evaluation activity. 

Since an evaluation study is a research activity, it is most useful to think in basic 
research terms which can then be applied to producing formative and summative 
evaluation reports. 

Research design easiiy reduces to a discussion of quantitative and qualitative 
methods which are used to answer very different and specific types of research 
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questions. Quantitative studies are normally used to test hypotheses that there are 
causal (or predictive) relationships between variables being examined. Qualitative 
studies, on the other hand, are usually exploratory in nature and are most useful for 
identifying variables that researchers be'ieve may be related. 

Based on the research question that is being asked, certain types of data must 
be collected under controlled/standardized conditions (the research model). 

According to the literature on research design, research models generally fall into two 
categories: experimental and quasi-experimental designs. For example, in testing a 
hypothesis (or conducting an evaluation of a program), an experimental design is the 
most powerful research model to use. In experimental designs, there is a comparison 
of an intervention group (experimental group) and a control group, on some outcome 
measure (usually measured quantitatively). There are ethical considerations when 
dealing with human subjects in an experimental condition. Well conducted quasi- 
experimental designs, however, can be quite powerful and are usually the choice in 
evaluation studies. 

Based on the type of data that is collected, certain statistical analyses become 
available for use. Statistical analyses range from basic descriptions to quite 
sophisticated multivariate procedures. Each statistical technique has assumptions, 
which if violated, render the results useless. Generally speaking, descriptive statistics, 
analysis of variance and chi-squares on nominal or ordinal data, or t-tests on interval 
or ratio data are sufficient for evaluation studies. These are powerful statistics when 
applied to data collected from a well designed and executed program (experiment). 
But still, discretion must be utilized in interpreting results that are statistically significant 
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but which may or may not be statistically meaningful. 

When proposing the development of a new educational program, it is important 
to document the need for the program, to detail the specific goals and objectives that 
will be met, to document who and how participants will be served, and to indicate 
some objective (or subjective) measure of success. A rigorous research design 
should include: clearly stated hypotheses and operationally defined variables; 
variables that are clearly manipulated and accurately measured by the researcher; and 
control or comparison groups to measure effects. An educational program plan 
should include clearly defined goals and objectives (information necessary for 
summative evaluation); and an implementation plan should specify project milestones 
(information necessary for formative evaluation). These are a priori activities to the 
implementation of the program. 

Proposed Research Models : 

In the proposed research models, the focus of the data collection activity and 
analysis is the classroom. Throughout the day, student movement from room to room 
across the campus is dynamic. In each classroom, students utilize available materials. 
The proposed research models assume that the educational technology is static to the 
classroom. Please see Exhibit 1 for presentation of suggested data items that may be 
utilized in a formative evaluation study. Exhibit 1 can be adjusted for use in situations 
where educational technology is not static to the classroom. 

In Part A, a classroom-level inventory of equipment not only points to age, 
functionality, and cost (plus repair and/or upgrade), but also allows for this information 
to be easily summarized to the campus and district levels. In Part B, the curriculum 
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overlay allows information regarding the courses taught in each of the classrooms, the 
degree of integration of technology into the curriculum, and actual student utilization of 
the technology to be cataloged and superimposed onto the inventory list in Part A. 

By overlaying the curriculum template onto the classroom-level inventory template, one 
can begin to identify patterns such as: in room X, the equipment is present but is not 
being used (this may point to needed changes in curriculum): whereas in room Y, the 
curriculum calls for the use of technology which is not available or adequate for use. 
The targeted data elements noted in Exhibit 1 may point to solutions as well. 
Equipment may need to be purchased, repaired or upgraded; there may be a need 
for staff training to help teachers further integrate technology into the curriculum: or it 
may turn out that classroom use might need to be rescheduled or that inventories 
might need to be reassigned. This is the stuff of formative evaluations. 

Please see Exhibit 2 for a presentation of suggested data items that may be 
utilized in a summative evaluation . In this example, a dropout reduction program for 
at-risk students intends to rely heavily on the use of educational technology for 
improving reading and math scores (as measured by a standardized test). Other 
program components such as participation in a mentoring program and increasing 
participation in school activities will serve to improve students’ self-esteem as well. 

Due to the combination of program activities, the student’s overall GPA will improve 
and students will be less likely to dropout. In this model, a quasi-experimental, 
pre/post-test only design was selected for use in researching the effectiveness of the 
program-one component of which was the use of educational technology. 

The targeted data items found in Exhibit 2 are based on the major program 







objectives used in the example. Based on the research design, data is collected, the 
program is implemented, and data is again collected at the end of the program. 
Descriptive statistics, analysis of variance and chi-squares on nominal data or ordinal 
data, or t-tests on interval or ratio data are then used (as indicated in the research 
design) in the analysis. 

Section Summary : 

The result of research-based models such as those outlined above can be 
used in formative and summative evaluation reports on the impact of educational 
technology. For example, an assessment of the effectiveness of the at-risk program 
that utilized educational technology as a program component can be folded into a 
report that addresses issues related to the overall use and integration of technology 
into the curriculum. The resulting report constitutes a formative and summative 
evaluation of the impact of educational technology. The data collection templates 
(found as exhibits) can be easily modified for individualized use in research/evaluation 
studies. 

Exhibits/Exercise : 

Exercise materials are found in a separate document to illustrate the use of 
Exhibits 1 and 2. For these materials, please contact the author. Exhibit 3 contains 
the template to be used in producing a research/program evaluation report on 
educational technology. 
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Exhibit 1 

Program Evaiuation Template 

Suggested Data Elements for Use in a Formative Evaluation of Educational Technology 



A-InvAntnry 


I iQt 




Rnnm* 


Equip. Purchase 


Repair/ 


Unit 


Total 


Code* Product Date 


Contract No. 


Cost 


Cost 


Computers: (specify configuration) 








Apple/Macintosh 






$ 


IBM and compatible equipment 






$ 


Integrated Learning Systems (specify) 






$ 


Peripherals and Inpirt/Output Devices: 








CD-Rom 






$ 


LaserDisks 






$ 


UPC Pens 






$ 


External Memory Storage 






$ 


Video Output (PC Vietwer) 






$ 


Hard Impact Printers 






$ 


Laser Printers 






$ 


Modem 






$ 


Other (specify) 






$ 


Software: fsoecifv application) 








Apple/Macintosh 






$ 


DOS 






$ 


Network (specify configuration) 






$ 


Other Educational Equipment: 








Audio Equipment 






$ 


Video Equipment 






$ 


Distance Learning Equipment 






$ 


TV Monitors 






$ 


Computer Furniture: 








Tables 






$ 


Desks 






$ 


Chairs 






$ 


Other Costs: 








Other (specify) 






$ 






Total: 


$ 


*Equipment code for inventory control and for use in Part B of this form. 






Part R-Ci irrini ill im nv/orlay 



Equip. 

Courses taught in this room: Code* 



Avq. No. 
Stds. 



Percent of 
Curriculum** 



Est. Use Total Percent 
by Stds.** Hours** of Total** 



*Specify educational technology utilized for each course taught in this room. 

**Separate handout contains the operational definitions and calculations for these data elements. 



For more information contact: J. E. Gonzalez at the Texas Center for Educational Research (800)580-8237. 



Exhibit 2 

Program Evaluation Template 

Suggested Data Elements for Use in Summative Evaluation of Educational Technology 
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For more information contact: J. E. Gonzalez at the Texas Center for Educational Research (600)580-8237. 



exhibit 3 

Program Evaluation Template 

Suggested Discussion Items for Use in Producing a Research/Program Evaluation Report on 
Educational Technology 



A research/program evaluation report should contain: 
Cover Page 

Program Evaluation Report Name: 
Program Evaluator: 

Date Report Submitted: 

Submitted to: 

For the Period: 



i. Executive Summary or Abstract 

Write this section last, but think about it first! II 
Use the “best line* from each of the following sections. 



ii. Table of Contents 



1. Background Information 

A. History of the program, needs analysis used as rationale for program. 

B. Goals/Objectives of the program. 

C. Program features (who and how served). 

D. Narrow the focus of the discussion to the items to be evaluated. 



2. Evaluation Study Description 

A. Identify the theoretical framework used as basis for the program. 

B. Discuss previous research studies used as models for the program. 

C. Describe the research protocol for the conduct of the study: 

1 . period of study 

2. measures and outcome measures 

3. specific statistical techniques 

4. potential weaknesses/limitations in the design, based on: 

theoretical framework for the program and previous research models. 



3. Results 

A. Results discussed in the context of the weaknesses/limitations in the design. 



4. Discussion of Results 

A. If the results were okay, and the design was okay, then it is plausible that 
effects seen are real effects: statistically significant and meaningful. 




5. Recommendations 

6. References/Resources 
Appendices 

For more information contact: J. E. Gonzalez at the Texas Center for Educational Research (S0O)58O-8237. 
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